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Surgical  Technology  Integration  with  Tools  for  Cognitive  Human  Factors 
(STITCH) 

2007  Annual  Report 


1.  Introduction 

The  need  for  accurate  assessment  in  surgical  training  has  become  even  more  important  with  the 
development  of  new  surgical  technologies,  many  of  which  have  transformed  methods  of 
treatment  for  both  the  patient  and  the  surgeon.  Difficult-to-master  technologies  such  as  the 
components  of  Minimally  Invasive  Surgery  (MIS)  highlight  the  need  for  surgical  competence 
but  do  not  inherently  provide  a  solution  for  how  to  define  and  measure  it.  The  long  term  goal  of 
this  research  is  to  build  an  integrated  surgical  technology  environment  designed  for  the 
continuous  monitoring  of  task  performance,  with  a  particular  focus  on  the  inclusion  of  important 
but  currently-overlooked  cognitive  measures. 

Evaluation  of  surgical  skill  in  MIS  can  be  made  more  accurate,  objective,  and  general  by 
considering  cognitive  and  environmental  factors  such  as  mental  workload,  stress,  situation 
awareness,  and  level  of  comfort  with  complex  tools.  This  research  will  show  that  a 
comprehensive  framework  for  measuring  cognitive  human  factors  in  MIS  settings  will  provide 
an  important,  statistically  significant  set  of  (largely  overlooked,  in  this  domain)  non-redundant 
metrics  for  evaluating  performance  in  the  context  of  new  technologies,  tasks,  and  learning 
methodologies. 

Software  development  efforts  will  produce  the  general-purpose  Plug-and-Play  (PnP)  framework 
and  application-specific  tools  usable  in  that  framework.  Well-defined  methodology  will  be 
incorporated  in  the  development  process  to  insure  required  safety,  reliability  and  robustness 
attributes  for  the  problem  domain.  The  human  studies  used  to  adapt,  select,  and  validate  the 
cognitive  measures  will  be  divided  into  four  parts:  an  equivalence  testing  phase  followed  by 
three  validity  studies.  The  equivalence  study  will  ensure  reliability  of  test  results  after  minor 
modifications.  The  subsequent  validity  tests  will  assess  construct  validity  (the  extent  to  which 
our  measures  of  stress  and  workload  dissociate  from  measures  of  performance  in  situations 
where,  theoretically,  they  should);  concurrent  validity  (sensitivity  to  differences  among  surgical 
conditions  known  to  differ  substantially  in  difficulty);  and  predictive  power  (degree  to  which 
measures  can  predict  more  complicated  indices  of  surgical  proficiency,  including  adaptive 
aspects  of  performance  on  which  students  have  not  received  explicit  training). 

The  STITCH  project  will  develop  a  specification,  design,  and  implementation  of  an  integrated 
surgical  training  and  assessment  framework  and  will  provide  assessment  results  for  specific 
cognitive  measures,  including  validity  and  predictive  studies.  These  results  will  be  useful  for 
implementing  improvements  in  training  methods  that  seek  to  use  valid  cognitive  measures  as 
part  of  the  assessment  strategy. 

2.  Major  Accomplishments 

In  this  section  we  provide  a  functional  view  of  major  tasks  accomplished  during  the  2007  project 
year.  These  include  (1)  timeline  in  obtaining  human  subject  approval,  on  which  the  cognitive 


experiments  depend;  (2)  progress  in  the  design  of  the  technical  architecture,  including  the  PnP; 
(3)  cognitive  human  factors;  and  (4)  the  production  of  research  media. 

2.1  Human  Subject  Approval 

This  project  began  on  September  26,  2006  and  involves  human  subjects  for  which  we  are 
requesting  institutional  IRB  approval.  This  approval  was  granted  by  the  University  of  Kentucky 
1RB  on  January  9,  2007.  The  protocol,  proposal  and  IRB  approval  were  then  forwarded  to  ORP 
for  final  approval.  On  7/25/07  we  received  ORP  approval  and  on  9/4/07  the  University  of 
Kentucky’s  IRB  approved  modifications  and  provided  finalized  consent  forms  for  enrolling 
participants.  More  detail  on  the  subjects  and  our  study  is  provided  below. 

Clinical  Collaboration 

We  have  met  once  per  week  at  the  Center  for  Minimally  Invasive  Surgery  in  the  Medical  School 
at  the  University  of  Kentucky  in  order  to  establish  and  supervise  the  research  plan  for  developing 
technology  and  an  experimental  plan  related  to  urological  surgical  training  and  intervention. 

The  Co-Pis  (Seales,  Carswell,  Strup)  have  used  these  meetings  for  planning  and  for  specific 
events  such  as: 

•  Direct  staff  to  set  up  experimental  stations  and  to  develop  videotaped,  narrated 
information  for  skills  performance  studies  and  to  support  subsequent  studies 

•  Use  MIS  conference  room  to  UK  OR  telesurgery  link  so  that  Dr.  Seales  and  Dr.  Carswell 
could  observe  two  live  remote  prostatectomy  procedures  performed  by  Dr.  Strup  in  OR  9 
of  the  UK  hospital.  This  telesurgery  session  included  two  way  audio  and  video  links  so 
that  pertinent  questions  that  Dr.  Seales  and  Dr.  Carswell  had  could  be  answered  by  Dr. 
Strup  in  real  time  while  he  conducted  the  procedure.  Two  cases  were  observed:  one  with 
a  robot  (Da Vinci)  and  one  as  straight  laparoscopy. 

•  Prepared  DVD  and  reviewed  copies  of  earlier  recorded  Narrated  Teaching  Case 
(Prostatectomy)  which  featured  interaction  between  Dr.  Strup  in  the  OR  and  Dr.  Seales 
and  Dr.  Carswell  in  the  MIS  conference  room.  Copies  were  technicians  so  that  they 
might  view  them  on  their  own  time  and  better  understand  the  ideas  and  concepts  that 
would  allow  advanced  imaging  techniques  to  increase  the  efficiency  of  selected  surgical 
procedures. 

2.2  System  Tools  and  Architecture 

PnP  Framework  Display  System.  The  flexible,  self-calibrating  display  framework  as 
developed  by  the  REVEAL  project  (and  deployed  and  working  at  UMMC)  is  a  single, 
monolithic  display  system  that  serves  a  single  application.  In  moving  forward  into  the 
integrated,  multi-tasked  PnP  environment,  we  need  to  generalize  the  framework  to  allow 
multiple  applications  to  render  their  output  onto  the  tiled  display  simultaneously.  This  presents  a 
number  of  challenges  in  terms  of  capturing  the  output  and  control  information  from  multiple 
applications,  to  the  display  of  these  applications  output  on  a  shared  display.  We  are  researching 
off-the-shelf  software  solutions  that  we  promise  to  allow  us  to  integrate  multiple  OpenGL 
streams  with  existing  window  management  software  with  minimal  additional  implementation 
effort.  The  goal  is  to  implement  a  seamless  OpenGL-based  compositing  window  manager 


extension,  which  preserves  the  flexible  tiled  display  characteristic  of  the  existing  system  but 
extends  the  system  to  allow  the  alignment  of  multiple  projectors  sharing  a  single  desktop  for 
high-resolution,  collaborative  displays.  One  member  of  the  STITCH  development  team  spent 
time  getting  training  in  the  use  of  LynxOS-178,  an  FAA-certified  commercial  real-time 
operating  system.  LynxOS  is  being  evaluated  for  the  implementation  of  a  high-reliability 
STITCH  framework. 

The  Stereo-Endoscopy  publication  ("Evaluation  of  Stereo-Endoscopy  for  Image  Field 
Reconstruction  in  Minimally  Invasive  Surgery”)  is  drafted  and  ready  for  publication  after  a 
second  round  of  data  collection  using  the  daVinci  Surgical  System  at  UK  Medical  Center.  The 
additional  data  will  allow  us  to  quantify  the  error  bounds  of  the  reconstructed  3D  points,  giving 
an  accurate  depth  range  for  which  anatomical  measurements  can  be  trusted.  The  tiled  display 
system  developed  under  Project  REVEAL  has  been  expanded  and  enhanced  greatly  in  the  past 
year.  Slow,  expensive  cluster-based  rendering  has  been  replaced  with  a  much  simpler  and  faster 
single  server  implementation  using  modem  COTS  hardware.  The  number  of  tiled  projectors  in 
our  display  was  expanded  from  eight  to  twelve,  bringing  total  available  pixel  resolution  to  more 
than  three  times  that  of  the  highest  resolution  HD  projectors.  The  display  system  can  now 
handle  both  standard  definition  and  high  definition  video  streams,  from  both  live  inputs  and  from 
disk.  It  functions  in  standard  two-dimensional  mode  as  well  as  three-dimensional  output  using 
polarized  lenses.  Output  capabilities  will  be  expanded  in  2008  to  allow  a  full  windowed 
environment  for  the  visualization  of  multiple  simultaneous  inputs  in  a  high-resolution  tiled, 
projected  display.  The  same  upgrades  were  rolled  out  to  our  mirror  site  at  the  University  of 
Maryland  Medical  Center’s  MASTRI  Center. 

We  have  modified  the  video  client  (a.k.a.  the  “Smartlmage”  video  client  system)  in  order  to 
prepare  it  for  integration  with  the  emerging  PnP  architecture.  It  has  been  overhauled  to  handle 
multiple  simultaneous  data  streams  (multiple  client  connections,  but  only  one  is  displayed  at  a 
time).  In  terms  of  computational  efficiency,  the  Smartlmage  client  used  only  a  small  portion  of  a 
large  and  relatively  slow  geometry  library  (CGAL)  to  determine  how  to  place  images  on  the 
display.  The  library  was  very  functional  for  a  first  prototype  but  included  cumbersome  overhead 
that  made  configuration  and  performance  issues  a  concern  for  our  leaner  production  version.  We 
replaced  that  library  with  a  much  smaller  and  faster  library  (GPC)  that  contained  little  more  than 
the  functionality  smartlmage  required.  This  reduced  the  size  of  the  Smartlmage  code  base  and 
deployed  executable  image,  made  Smartlmage  compile  five  times  faster,  and  the  new  library 
seamlessly  integrates  into  the  Smartlmage  architecture.  Additionally,  a  calibration  speed 
improvement  should  be  noticed  on  larger  displays  because  the  new  library  requires  far  fewer 
function  calls  and  implements  several  functions  through  C  macros  rather  than  C++  class 
methods. 

Using  this  software  we  have  the  capability  to  play  captured  video  and  to  display  video  from  a 
DVD  (2D  and  3D).  The  performance  is  measured  at  30fps  @1 10ms  latency  video  (including 
stereo  video).  We  have  worked  to  integrate  HD  results  from  the  REVEAL  project  in  order  to 
integrate  within  the  PnP  framework,  including  HD  capture  capabilities  with  AJA  card,  integrated 
with  the  Stryker  1088  HD  scope. 


Endoscopic  Tools  Highlights.  Our  video  capabilities  were  greatly  enhanced  with  the 
integration  of  a  Stryker  1088  HD  endoscope  into  our  display  system.  We  can  capture  and 
display  full  frame-rate  1280x1024  video.  Overall  display  latency  is  around  200  milliseconds, 
near  the  upper  bound  of  the  acceptable  limit  for  interactivity. 

We  completed  calibration  of  two  stereoscopic  endoscopes  for  the  assessment  of  taking  intra¬ 
operative  metric  measurements.  Thanks  to  collaboration  with  Dr.  Stephen  Strup  of  the 
University  of  Kentucky  Department  of  Surgery,  we  gained  access  to  the  daVinci  Surgical 
System,  a  robotic  surgical  platform,  to  capture  calibration  data  for  its  stereo  scope.  We 
compared  those  results  to  that  of  an  older  Vista  StereoScope  in  our  lab.  Our  previous  results 
with  the  Vista  had  been  nearly  unusable,  but  the  daVinci  robot  showed  a  four-fold  improvement 
in  measurement  accuracy.  This  illustrates  that  calibration  of  high-distortion  endoscopes  with  a 
pin-hole  camera  model  is  clearly  possible,  and  useful  data  may  come  of  this  later.  Given  the 
proper  constraints,  intra-operative  measurement  may  be  possible  and  warrants  future  study. 

Inertial  endoscopic  tracking.  We  are  evaluating  the  use  of  an  inertial  measurement  unit  (1MU) 
to  track  the  endoscope  for  improved  image  reconstruction.  We  purchased  a  MicroStrain  Inertia- 
Link  for  this  purpose  and  are  using  software  developed  by  Fremont  Associates  to  integrate  it 
with  our  system.  A  prototype  system  should  be  ready  in  1-2  months.  We  hope  that  this  will  be 
the  first  of  many  instruments  to  fully  integrate  with  the  STITCH  software  platform  currently 
being  implemented. 

Faro  reconstruction.  We  have  collaborated  with  the  Smartlmage  project  to  develop  a  method 
for  accurate  3D  reconstruction  using  through-the-scope  video.  The  FaroArm,  a  coordinate¬ 
measuring  machine  with  laser  scanner,  can  scan  objects  in  3D  to  produce  ground  truth  data  for 
image  registration.  It  can  also  be  attached  to  an  endoscope  to  provide  extremely  accurate 
tracking  for  the  best  possible  structure-from-motion  computer  vision  techniques. 

PnP  Framework  Requirements 

We  have  installed  and  begun  using  the  Telelogic  DOORS  (Dynamic  Object  Oriented 
Requirements  System)  requirements  management  system  to  manage  project  requirements.  The 
software  was  received  and  installed,  and  staff  has  completed  training  on  the  deployment  and  use 
of  the  system. 

With  our  tools  framework  in  place,  we  have  begun  identifying  and  documenting  end-user  and 
system  requirements  for  the  PnP  Framework  system.  The  requirements  identified  to  date  have 
included  (1)  fundamental  goals  for  the  system — i.e.,  its  intended  audience  and  use;  (2)  basic 
architectural  components  of  the  system  including  input  stream  types,  outline  of  a  passive 
information  repository,  and  output  stream  and  display  requirements;  and  (3)  basic  scenarios  for 
the  use  of  the  system  in  clinical  research  settings. 

Framework  Input  Modules 

Sensors  to  augment  traditional  image  streams  are  one  of  the  critical  input  stream  components  for 
the  PnP  framework.  Our  first  sensor  implementation  will  be  based  on  a  temperature- 
compensated  six  degree-of-freedom  inertial  measurement  unit  (IMU).  We  acquired  equipment 
and  have  begun  testing,  to  include  the  creation  of  software  infrastructure  to  communicate  with 
the  device  from  within  the  STITCH  framework. 


The  formalization  of  an  initial  user  requirements  document  was  completed,  based  on 
requirements  embodied  in  the  initial  STITCH  project  proposal.  These  requirements  describe, 
from  the  user’s  point  of  view,  required  and  desirable  attributes  of  the  system  to  be  engineered. 

PnP  framework  input  modules  will  initially  encompass  (1)  NTSC  video,  (2)  synchronized  HD 
audio/video,  and  (3)  high-resolution  location  information  derived  from  a  six  degree-of-freedom 
inertial  measurement  unit  (IMU).  Video  input  modules  will  be  derived  from  the  VIBE  system  of 
project  REVEAL.  The  IMU  input  module  is  new  development  being  carried  out  under  the 
STITCH  project. 

During  the  year  significant  progress  was  made  on  the  design,  implementation  and  test  of  the 
IMU  input  module.  The  current  design  supports  two  modes  of  operation — a  synchronous 
procedure-call  mechanism  that  will  return  one  measurement  for  each  call,  and  an  asynchronous 
callback  mechanism  that  will  be  driven  by  the  IMU  device’s  maximum  sampling  rate.  The 
design  being  implemented  to  support  these  two  modes  of  operation  is  shown  in  Figure  1. 


(ADT  To  User  via  Callback  Function) 
Figure  1:  IMU  API  System  Architecture 


We  have  implemented  the  synchronous  procedure-call  mechanism,  which  includes  the 
completion  of  all  design,  coding,  verification  and  validation  activities.  Testing  demonstrated 
that  the  implemented  API  functionality  correctly  provided  single-precision  linear  acceleration 
and  angular  rate  data  to  calling  programs  using  the  matrix  abstract  data  type  of  the  BOOST 
template  library. 

IMU  API:  The  design,  implementation  and  system-level  test  of  the  IMU  API  was  completed 
during  the  quarter,  and  the  software  was  deployed  for  use  by  UK  CVVM  staff  of  STITCH.  The 


completion  of  the  API  library  will  support  implementation  of  the  first  PnP  framework  input 
device  based  on  inertial  tracking  of  a  COTS  laparoscope.  The  1MU  API  deliverable  was 
transferred  from  Fremont  Associates  and  their  technical  staff  helped  the  STITCH  team  at  UK 
begin  integration  with  STITCH  framework.  We  are  currently  developing  3D  tracking  software 
using  inertial  measurement  to  improve  endoscopic  reconstruction  techniques.  Preliminary 
studies  of  this  approach  were  prepared  for  the  imminent  Q4  2007  (October  2007)  site  visit  from 
TATRC  personnel. 

Phase  I  Protocol  Support  Technology  and  Collection  Framework:  STITCH  development 
team  members  are  working  with  Dr.  Carswell’s  team  to  automate  much  of  the  data  collection 
currently  required  by  the  cognitive  task  studies.  At  present,  task  complexity  measures  and 
subject  time  estimations  are  recorded  manually,  by  a  research  assistant  observing  the 
performance  of  the  task.  During  the  year  we  have  been  researching  and  testing 
software/hardware  solutions  that  will  allow  this  data  to  be  captured  by  software,  directly  from 
the  subject's  verbal  utterances.  Based  on  this  work  we  have  begun  implementation  of  speech 
recognition  capabilities  to  improve  automation  and  decrease  workload  of  human  factors  studies. 
This  will  allow  for  more  subjects  to  be  processed  with  less  data  entry  and  quicker  analysis. 

Tools  for  Prostatectomy 

In  order  to  test  a  new  visualization  and  navigation  tool  for  use  in  prostatectomy  cases  that 
present  ambiguous  geometries  and  difficult  approach  angles  we  have  developed  software  for 
accurate  feature-based  texture  mapping  onto  3D  geometry.  The  first  test  of  the  algorithms  on 
which  this  software  is  based  uses  a  FaroArm  3D  laser-based  scanner  to  capture  geometry  and  a 
high  resolution  digital  camera  to  capture  2D  images.  The  3D  texture  registration  problem  is  the 
correct  application  of  texture  coordinates  to  a  captured  3D  mesh.  The  goal  in  the  case  of  the 
surgical  procedure  is  to  orient  the  high  resolution  scope  view  with  respect  to  anatomical 
fiducials,  pre-opertative  CT  scans,  and  known  angles  established  at  the  start  of  the  surgical 
procedure.  Registration  can  aid  in  orientation  and  identification  of  crucial  anatomical  structure. 
In  our  current  implementation  we  work  with  data  that  is  a  combination  of  untextured  3D  point 
sets  from  a  Faro  Laser  ScanArm  and  high-resolution  texture  images.  In  order  to  place  the  texture 
onto  the  3D  mesh,  we  perform  a  calibration  procedure  which  gives  us  the  intrinsic  (i.e.  focal 
length  and  lens  distortion)  and  extrinsic  (relationship  between  2D  pixel  coordinates  in  the 
camera  and  3D  coordinates  in  the  ScanArm)  properties  of  the  camera.  These  properties  allow  us 
to  perform  a  projective  transform  to  map  the  texture  onto  the  mesh. 

One  of  the  problems  we  are  working  on  is  using  a  calibrated  texture  and  model  pair  to  map 
uncalibrated  textures  onto  the  same  model.  In  the  data  we  are  working  with,  the  uncalibrated 
image  is  assumed  to  be  of  the  same  object.  However,  the  object  may  be  deformed  differently 
from  the  captured  model  (i.e.  non-rigidly  deformed  during  the  surgery).  We  have  implemented 
an  algorithm  which  uses  template  matching  to  compute  reliable  point  correspondences  between 
the  images,  then  performs  a  simple  rotation  and  translation  based  on  these  correspondences.  If 
the  model  had  only  undergone  rotation  and  translation  with  respect  to  the  camera,  this  would  be 
sufficient,  but  we  must  also  deal  with  non-uniform  distortion  due  to  the  object's  possible 
deformation.  This  is  done  by  first  finding  strong  feature  points  (as  determined  by  minimum 
gradient  eigenvalues)  in  the  calibrated  image,  then  finding  their  corresponding  points  in  the 
uncalibrated  image  using  an  optical  flow  algorithm.  The  points  are  inserted  into  a  Delaunay 


triangulation,  and  then  a  triangulation  warp  is  applied  to  the  uncalibrated  image  in  order  to  place 
the  features  in  to  the  same  positions  as  the  calibrated  image.  Using  this  method  it  is  possible  to 
use  a  calibrated  image  to  map  uncalibrated  images  with  desirable  features  (i.e.  enhanced 
resolution,  improved  lighting)  onto  the  previously  acquired  3D  mesh.  Initial  testing  of  this 
algorithm  has  yielded  good  results  which  are  currently  being  used  as  a  benchmark  for  a  testing 
review.  We  are  working  on  making  the  algorithm  to  be  scale-independent  (in  the  current 
algorithm,  the  model  may  be  deformed  but  it  is  assumed  to  be  approximately  the  same  distance 
from  the  camera). 

In  terms  of  the  visual  presentation  of  this  process  and  its  computational  requirements,  we  have 
started  an  analysis  as  a  function  of  desired  image  resolution  and  available  computational  cycles. 
Due  to  the  high  resolution  of  the  textures  used  (39  megapixels),  special  techniques  must  be 
employed  in  order  to  display  the  textured  model  using  standard  graphics  hardware.  Currently  the 
texture  is  split  into  multiple  textures  with  an  overlapping  border,  such  that  each  polygon  is  fully 
contained  by  at  least  one  split  texture.  Optimizations  for  this  splitting  and  rendering  process  are 
underway  so  that  it  requires  less  texture  memory  and  renders  faster,  as  well  as  allowing  for 
optimal  display  of  the  texture  on  machines  which  cannot  fit  the  entire  texture  into  memory. 

2.3  Cognitive  Human  Factors 

Our  activities  during  Year  1  were  targeted  at  fundamental  psychometric  issues  (e.g.,  reliability 
and  validity  studies)  necessary  for  the  development  of  a  cognitive  human  factors  toolkit  for 
laparoscopy.  The  overall  aim  is  to  develop  a  toolkit  that  will  be  integrated  into  our  surgical 
technology  framework  for  use  in  assessing  surgical  training  methods,  new  surgical  technologies, 
and  even  surgical  expertise.  The  activities  of  Year  1  included  facility  and  instrumentation 
development,  research  protocol  development,  and  actual  data  collection. 

Facility/Instrumentation  Development.  Our  laboratory  space  and  resources  were  enlarged 
during  the  current  research  cycle  in  order  to  allow  us  to  collect  new  types  of  behavioral  data 
(eye-tracking)  and  to  increase  our  ability  to  complete  participant-running  and  multiple  user 
studies  in  parallel.  The  improvements  include  the  following: 

•  Completion  of  a  secondary  CVVE  usability  lab.  In  order  to  free  up  the  main  surgical 
simulation  system  at  the  CVVE  for  software  development  and  testing,  a  second 
laboratory  was  equipped  to  support  participant-running  using  a  training  stand  and 
equipment  similar  to  those  used  in  the  MIS  Center  on  the  UK  Medical  Center.  The  new 
lab  includes  a  Stryker  888  laparoscope,  surgical  trainer,  and  Maryland  dissectors,  Sony 
19-  inch  monitor,  Stryker  Quantum  3000  light  source,  and  Stryker  888  video  camera. 

•  Completion  of  main  campus  usability  lab.  This  lab  contains  equipment  equivalent  to 
that  in  the  CVVE  and  MIS  facilities,  and  it  is  housed  in  the  UK  Psychology  Department’s 
research  annex  on  the  UK  main  campus.  This  lab  allows  us  to  more  easily  recruit 
research  participants  from  the  PSY  100  subject  pool  who  sometimes  have  difficulty 
traveling  downtown  to  participate  in  research  at  the  CVVE. 


Figure  2:  The  Usability  lab  at 
the  Center  for  Visualization 


Figure  3.  Stereo-Head  and  digital  cameras  with  Classic 
lenses  attached. 


•  Delivery  of  FaceLAB.  We  received  our  FaceLAB  4. 1  eye-tracking  system  during  the 
last  quarter  of  2007.  Since  that  time,  we  have  worked  on  the  setup  of  the  system  in  the 
main  CVVE  lab,  and  we  have  collected  pilot  data  to  demonstrate  the  use  of  the  system. 
The  following  images  shows  the  arrangement  of  the  device  when  used  by  a  UK  surgeon 
in  the  lab  context,  as  well  as  a  screen  shot  of  the  user  interface  showing  a  variety  of  real¬ 
time  blink  measures,  saccade  measures,  and  position  tracking. 


Figure  4.  Four  displayed  windows  of  the  FaceLAB:  The  Main  Window,  the 
Control  Window,  the  Video  Window,  and  the  World  Model  Window  (clockwise). 


•  Eye  tracker  training.  Core  members  of  the  research  team,  including  Melody  Carswell, 
Matt  Field,  Cindy  Lio,  and  Qiong  Han,  completed  two  days  of  training  with  a  FaceLAB 
engineer  who  visited  our  main  CVVE  lab  on  December  3rd  and  4th.  The  training  allowed 
us  to  find  ways  to  record  a  wider  variety  of  participants  (e.g.,  those  with  and  without 
glasses,  with  different  iris/pupil  contrast  levels,  etc.).  It  also  allowed  us  to  acquire 
information  necessary  for  the  successful  integration  of  FaceLAB  and  STITCH  software. 


Protocol  Development 


•  IRB/HSRO  approval.  The  following  is  a  timeline  describing  our  progress  in 
developing  and  testing  the  subjective  measures  for  the  toolkit: 

al  human  subjects  review  completed,  and  approval  given,  by  UK  1RB. 

2/07  -  UK  approval  and  protocol  materials  sent  to  Army  ORP 

4/07  -  Initial  review  by  contact  officer. 

5/07  -  Updated  PI  training  completed  and  full  administrative  review  by  ORP  initiatied. 
6/07  -  Full  review  feedback  received. 

7/18/07  -  Response  and  modification  of  protocol  by  Pis. 

7/24/07  -  Approval  granted  by  ORP,  pending  approval  of  required  modifications  by  UK 
IRB. 

9/4/07  -  UK  IRB  approves  modifications  and  provides  finalized  consent  forms  for 
enrolling  participants. 

9/4/07  -  9/7/07  ~  over  1400  PSY  100  participants  were  screened  for  suitability  for 
enrollment  in  our  studies  in  a  series  of  large-scale  screening  sessions.  Our  inclusion 
criteria  give  preference  to  students  who  are  preparing  for  employment  in  health-care 
fields  and  who  report  playing  video  games  on  a  regular  basis. 

9/21/07  -  Changes  completed  at  the  Usability  Lab  at  the  CVVE  to  allow  participant¬ 
running  in  an  area  apart  from  main  STITCH  programming/development. 

9/24/07  -  9/28/07  -  3  pilot  subjects  run  to  estimate  number  of  trials  per  participant  that 
can  be  completed  within  allotted  time. 

10/10/07  -  14  enrolled  and  run  through  full  protocol. 

•  Continuation  review  in  process.  The  IRB  continuation  review  materials  were 
submitted  on  November  26,  2007  and  will  be  reviewed  during  the  week  of  December  17, 
2007.  Modifications  that  were  made  included: 

(a)  Personnel  list.  Four  Psychology  undergraduate  students  who  will  be  conducting  the 
study  as  their  partial  fulfillment  of  an  independent  research  course,  and  the 
Information  Technology  Manager  at  MIS  were  included  in  the  continuation  review 
personnel  list. 


(b)  Address  change.  Since  our  offices  and  labs  are  at  the  Center  for  Visualization  and 
Virtual  Environments,  we  requested  to  receive  future  IRB  notifications  at  1  Quality 
Street,  Suite  800,  Lexington,  KY  40507-1464. 

•  Training  procedure.  The  8-minute  training  involved  showing  to  the  participants  the 
three  tasks.  We  explained  the  operation  of  the  dissectors,  the  setting,  and  the  three  tasks 
in  the  beginning  of  the  session.  The  participants  then  practice  each  of  the  tasks  in  open 
view. 

•  Task  Development.  The  following  three  tasks  were  used  as  experimental  tasks  in  an 
effort  to  be  consistent  with  training  tasks  currently  in  use  at  the  UK  MIS  Training  Center. 

The  goal  of  the  pegboard  task  for  the  participant  is  to  transfer  as  many  rings  as  possible 
from  a  small  dish  and  place  them  on  a  pegboard  (left)  They  are  to  transfer  as  many  as 
they  can  per  unit  time  (90  seconds  for  this  study).  For  the  cobra  rope  task,  the  goal  is  to 
“inspect”  the  rope  by  transferring  it  with  two  graspers  down  its  entire  length  as  quickly  as 
possible  while  only  touching  sections  of  rope  that  are  a  specific  color  (center).  For  the 
cannulation  task,  the  goal  is  to  thread  a  pipe  cleaner  through  a  plastic  tube  as  quickly  as 
possible  without  bending  the  pipe  cleaner  (right). 


Figure  5:  As  seen  through  the  laparoscope,  from  left  to  right:  Pegboard  task,  cobra  rope  task,  and  cannulation  task. 


•  Modification  of  Questionnaires.  We  modified  the  verbal  questionnaires  after 
conducting  the  study  with  three  Psychology  students.  The  digitized  questionnaires  that 
we  developed  previously  took  significantly  longer  to  complete  than  the  written  version. 
Consequently,  for  the  verbal  protocol,  we  explained  to  the  participants  the  questionnaires 
and  scoring  system  (zero  to  100)  at  the  end  of  first  trial.  In  the  remaining  trials,  we  read 
the  NASA-TLX,  NASA-TLX  paired  comparisons,  and  MRQ  questionnaires  to  the 
participants  without  description.  We  also  read  the  SSSQ  to  the  participants  at  the  end  of 
each  trial  of  the  last  block. 

Data  Collection 

•  Reliability  Study.  We  collected  data  at  both  Breckinridge  on  campus  and  at  the  Center 
for  Visualization  and  Virtual  Environments  (CVVE).  A  total  of  sixty-eight  PSY  100 
students  participated  in  the  study,  of  which  21  of  took  place  at  the  CVVE  and  48  at 
Breckinridge.  Written  and  verbal  conditions  were  assigned  by  an  odd  and  even  numbers 


system  across  participants  while  tasks  were  randomized  within  each  block  (a  total  of  5 
blocks).  After  completing  each  task,  the  participants  answered  the  questionnaires 
according  to  their  assigned  protocol  (visual/written  vs.  auditory/vocal  versions). 

•  More  than  half  of  the  data  have  been  coded  and  entered  into  our  database  and  very 
preliminary  analyses  have  been  completed.  We  are  finding  that  only  some  subscales  of 
the  NASA-TLX  have  adequate  reliability  (e.g.,  “mental  demand”  but  not  “temporal 
demand”);  however,  these  reliability  estimates  have  not  yet  been  corrected  for 
experimenter  effects  (i.e.,  slight  differences  in  the  way  different  experimenters  might 
have  administered  the  materials).  A  more  extensive  analysis  will  be  completed  when  the 
remaining  data  are  entered,  at  which  point  we  can  also  ask  questions  about  relative 
correlations  of  each  administration  method  and  data  from  performance  measures  (both 
primary  task  and  secondary  task  performance). 

2.4  Production  of  Research  Media 

This  year  we  have  systematically  captured  detailed  research  footage,  including  methods, 
systems,  and  interviews,  in  order  to  produce  a  findings  documentary  that  will  detail  methods, 
approaches,  systems,  and  findings.  This  is  currently  in  production  and  we  expect  to  complete  the 
project  in  two  phases,  the  first  finished  in  mid-2008  and  the  second  at  project  year  2  completion. 


3.  Project  Milestones  2007 

The  2007  project  plan  led  us  to  near  completion  of  visualization  and  display  technology 
development,  moving  the  technical  component  of  the  project  toward  a  support  role  for  remaining 
ergonomic  assessment  activities  at  the  University  of  Maryland  medical  center.  We  have  been 
granted  a  no-cost  extension  to  complete  the  ergonomic  assessment,  with  the  technical  team 
providing  support  for  the  equipment  as  the  protocol  is  executed.  The  accomplished  milestones 
and  our  progress  in  reaching  each  for  the  year  are  assessed  in  the  sections  that  follow. 

Project  Milestones 

This  study  was  designed  for  one  year,  although  we  have  made  progress  and  expect  to  use  second 
year  funds  for  continued  development  and  experimentation.  We  will  work  to  complete  technical 
milestones  in  each  of  the  following  areas: 

1)  Design  and  Build  First  Generation  PnP  Framework 

a)  Identify  and  document  requirements  from  various  constituencies  including  clinical, 
training,  evaluation,  peripheral  devices,  MD  PnP.  (Months  1-4). 

b)  Define  and  document  functional  and  structural  architecture  for  framework.  Define  data 
architecture  in  terms  of  XML  data  streams  and  SQL  database  structures.  (Months  3-6). 

c)  Input  Modules  (Sensors,  positioners,  etc.) 

i)  Design,  documentation  and  review.  (Month  7) 

ii)  Implementation  and  unit  test.  (Months  7-9). 

d)  Processing  Modules  (API  and  implementation  for  “toolkit”  of  processing  methods.) 

i)  Design,  documentation  and  review.  (Month  9). 

ii)  Implementation  and  unit  test.  (Months  9-11). 


e)  Output  Modules  (Displays,  effectors,  etc.) 

i)  Design,  documentation  and  review.  (Month  8). 

ii)  Implementation  and  unit  test.  (Months  8-10). 

j)  Archive  and  Query  Modules  (Underlying  database  and  query  engine.) 

i)  Design,  documentation  and  review.  (Month  7). 

ii)  Implementation  and  unit  test.  (Months  7-9). 

g)  Integration  and  Testing.  (Month  11-12). 

h)  Deployment.  (Month  12). 

2)  Design,  Implement,  and  Test  Advanced  Surgical  Tools 

a)  Design  technical  framework  for  potential  solutions  (simulation/lab  tests  -  Months  1  -4) 

b)  Consult  during  design  phase  with  UMMC  experts  (Months  1-3) 

c)  Deploy  for  testing  by  experts  at  UMMC  (Months  5-12) 

d)  Revise  implementations  as  required  (Months  6-12) 

e)  Prepare  for  testing  in  cognitive  frameworks  (Months  10-12) 

3)  Design  Cog  Monitoring  Protocols 

a)  Complete  the  adaptation  of  secondary  task  (time  estimation)  and  subjective  workload 
measures  to  make  them  appropriate  for  the  surgical  context  and  to  allow  for  integration 
of  administration  and  analysis  functions  into  the  proposed  continuous  performance 
monitoring  system.  (Months  3  to  5.) 

b)  Provide  preliminary  evaluation  of  the  use  of  eye  blink  and  pupil  size  measures  of  mental 
workload  in  a  training  context  with  possible  integration  of  these  measures  into  the 
continuous  performance  monitoring  system.  (Months  3  to  5.) 

4)  Design  and  Execute  Human  Study 

a)  Complete  human  subjects  approval  and  initiate  subject  recruitment.  (Months  1  and  2.) 

b)  Execute  human  studies  of  construct,  concurrent,  and  predictive  validity  within  the  lab  and 
training  environment.  (Months  6  to  12.) 

c)  Conduct  user  evaluation  of  use  of  surgical  robot  (v.v.  conventional  procedures),  with  and 
without  on-demand  tools,  using  our  new  suite  of  cognitive  metrics.  (Months  13  to  24.) 

Project  Conclusions  and  Deliverables 

The  project  has  received  Year  2  funding  and  will  extend  reported  results.  Final  conclusions  will 
be  reported  in  the  final  report  at  the  end  of  Year  2.  We  have  made  available  all  source  code 
detailed  for  the  technical  environments  and  algorithms  above,  including  technical  specifications 
for  hardware  and  configuration  information  where  necessary.  In  addition,  we  have  published 
papers  and  made  presentations  as  detailed  in  the  reference  list.  One  pre-publication  paper  is 
included  in  the  appendix  for  reference. 
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